{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K8HAOWyDgPiH"
   },
   "source": [
    "# Shallow methods for supervised learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v6w1KA97gXao"
   },
   "source": [
    "In this notebook we will exploring a very naive (yet powerful) approach for solving graph-based supervised machine learning. The idea rely on the classic machine learning approach of handcrafted feature extraction.\n",
    "\n",
    "In Chapter 1 you learned how local and global graph properties can be extracted from graphs. Those properties represent the graph itself and bring important informations which can be useful for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "5k3sYIRJpMgb",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uninstalling stellargraph-1.2.1:\r\n",
      "  Successfully uninstalled stellargraph-1.2.1\r\n"
     ]
    }
   ],
   "source": [
    "!pip install stellargraph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yWL_AuChPcYS"
   },
   "source": [
    "In this demo, we will be using the PROTEINS dataset, already integrated in StellarGraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "id": "gS5B47T2gWll",
    "outputId": "4020adc2-75b7-4aa5-b480-24d2693a8a74"
   },
   "outputs": [],
   "source": [
    "from stellargraph import datasets\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "dataset = datasets.PROTEINS()\n",
    "display(HTML(dataset.description))\n",
    "graphs, graph_labels = dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDlUMUUFLrjh"
   },
   "source": [
    "To compute the graph metrics, one way is to retrieve the adjacency matrix representation of each graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "qsOw9zFwrxDe"
   },
   "outputs": [],
   "source": [
    "# convert graphs from StellarGraph format to numpy adj matrices\n",
    "adjs = [graph.to_adjacency_matrix().A for graph in graphs]\n",
    "# convert labes fom Pandas.Series to numpy array\n",
    "labels = graph_labels.to_numpy(dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "6S5M5mL2t-ik"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import networkx as nx\n",
    "\n",
    "metrics = []\n",
    "for adj in adjs:\n",
    "    G = nx.from_numpy_matrix(adj)\n",
    "    # basic properties\n",
    "    num_edges = G.number_of_edges()\n",
    "    # clustering measures\n",
    "    cc = nx.average_clustering(G)\n",
    "    # measure of efficiency\n",
    "    eff = nx.global_efficiency(G)\n",
    "\n",
    "    metrics.append([num_edges, cc, eff])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1_a5CiZKL4vW"
   },
   "source": [
    "We can now exploit scikit-learn utilities to create a train and test set. In our experiments, we will be using 70% of the dataset as training set and the remaining as testset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NRrNPqOxu7eY"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(metrics, labels, test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TMIF1weiMO0F"
   },
   "source": [
    "As commonly done in many Machine Learning workflows, we preprocess features to have zero mean and unit standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9qUjNhPru6ni"
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_train)\n",
    "\n",
    "X_train_scaled = scaler.transform(X_train)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QqaZzejRMdmu"
   },
   "source": [
    "It's now time for training a proper algorithm. We chose a support vector machine for this task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "L3A6_fh0OV9x",
    "outputId": "6297d8fe-3cc9-435b-e8fe-b50425aaee24"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.7455089820359282\n",
      "Precision 0.7709251101321586\n",
      "Recall 0.8413461538461539\n",
      "F1-score 0.8045977011494253\n"
     ]
    }
   ],
   "source": [
    "from sklearn import svm\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "\n",
    "clf = svm.SVC()\n",
    "clf.fit(X_train_scaled, y_train)\n",
    "\n",
    "y_pred = clf.predict(X_test_scaled)\n",
    "\n",
    "print('Accuracy', accuracy_score(y_test,y_pred))\n",
    "print('Precision', precision_score(y_test,y_pred))\n",
    "print('Recall', recall_score(y_test,y_pred))\n",
    "print('F1-score', f1_score(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NBVKcDWHeGoR"
   },
   "source": [
    "# Supervised graph representation learning using Graph ConvNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lb6FvAQ3eUNs"
   },
   "source": [
    "In this notebook we will be performing supervised graph representation learning using Deep Graph ConvNet as encoder.\n",
    "\n",
    "The model embeds a graph by using stacked Graph ConvNet layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VHU1UGiHfw1e"
   },
   "source": [
    "In this demo, we will be using the PROTEINS dataset, already integrated in StellarGraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "id": "_8SDtHy1PfNx",
    "outputId": "aa1f8875-ab8d-42b6-eadc-8084f1796cc1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Each graph represents a protein and graph labels represent whether they are are enzymes or non-enzymes. The dataset includes 1113 graphs with 39 nodes and 73 edges on average for each graph. Graph nodes have 4 attributes (including a one-hot encoding of their label), and each graph is labelled as belonging to 1 of 2 classes."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from stellargraph import datasets\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "dataset = datasets.PROTEINS()\n",
    "display(HTML(dataset.description))\n",
    "graphs, graph_labels = dataset.load()\n",
    "\n",
    "labels = graph_labels.to_numpy(dtype=int)\n",
    "\n",
    "# necessary for converting default string labels to int\n",
    "graph_labels = pd.get_dummies(graph_labels, drop_first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_uEUzYBIM6cK"
   },
   "source": [
    "StellarGraph we are using for building the model, uses tf.Keras as backend. According to its specific, we need a data generator for feeding the model. For supervised graph classification, we create an instance of StellarGraph's PaddedGraphGenerator class. This generator supplies the features arrays and the adjacency matrices to a mini-batch Keras graph classification model. Differences in the number of nodes are resolved by padding each batch of features and adjacency matrices, and supplying a boolean mask indicating which are valid and which are padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "i34QgSA_P_sM"
   },
   "outputs": [],
   "source": [
    "from stellargraph.mapper import PaddedGraphGenerator\n",
    "generator = PaddedGraphGenerator(graphs=graphs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YepZYuk_NWWT"
   },
   "source": [
    "Now we are ready for actually create the model. The GCN layers will be created and stacked togheter through StellarGraph's utility function. This _backbone_ will be then concateneted to 1D Convolutional layers and Fully connected layers using tf.Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "qF8DHIalQuIW"
   },
   "outputs": [],
   "source": [
    "from stellargraph.layer import DeepGraphCNN\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.layers import Dense, Conv1D, MaxPool1D, Dropout, Flatten\n",
    "from tensorflow.keras.losses import binary_crossentropy\n",
    "import tensorflow as tf\n",
    "\n",
    "nrows = 35  # the number of rows for the output tensor\n",
    "layer_dims = [32, 32, 32, 1]\n",
    "\n",
    "dgcnn_model = DeepGraphCNN(\n",
    "    layer_sizes=layer_dims,\n",
    "    activations=[\"tanh\", \"tanh\", \"tanh\", \"tanh\"],\n",
    "    k=nrows,\n",
    "    bias=False,\n",
    "    generator=generator,\n",
    ")\n",
    "gnn_inp, gnn_out = dgcnn_model.in_out_tensors()\n",
    "\n",
    "\n",
    "x_out = Conv1D(filters=16, kernel_size=sum(layer_dims), strides=sum(layer_dims))(gnn_out)\n",
    "x_out = MaxPool1D(pool_size=2)(x_out)\n",
    "\n",
    "x_out = Conv1D(filters=32, kernel_size=5, strides=1)(x_out)\n",
    "\n",
    "x_out = Flatten()(x_out)\n",
    "\n",
    "x_out = Dense(units=128, activation=\"relu\")(x_out)\n",
    "x_out = Dropout(rate=0.5)(x_out)\n",
    "\n",
    "predictions = Dense(units=1, activation=\"sigmoid\")(x_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZOj3TjPIN4ev"
   },
   "source": [
    "Let's now compile the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "clWqCmfLJjBF"
   },
   "outputs": [],
   "source": [
    "model = Model(inputs=gnn_inp, outputs=predictions)\n",
    "model.compile(optimizer=Adam(lr=0.0001), loss=binary_crossentropy, metrics=[\"acc\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ZnhaSMDN9ii"
   },
   "source": [
    "We use 70% of the dataset for training and the remaining for test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "j3Hr6_FyJ5m4"
   },
   "outputs": [],
   "source": [
    "from sklearn import model_selection\n",
    "train_graphs, test_graphs = model_selection.train_test_split(\n",
    "    graph_labels, test_size=.3, stratify=labels,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "p9_2ybPqJ-3B"
   },
   "outputs": [],
   "source": [
    "gen = PaddedGraphGenerator(graphs=graphs)\n",
    "\n",
    "train_gen = gen.flow(\n",
    "    list(train_graphs.index - 1),\n",
    "    targets=train_graphs.values,\n",
    "    symmetric_normalization=False,\n",
    "    batch_size=50,\n",
    ")\n",
    "\n",
    "test_gen = gen.flow(\n",
    "    list(test_graphs.index - 1),\n",
    "    targets=test_graphs.values,\n",
    "    symmetric_normalization=False,\n",
    "    batch_size=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wCNr8_IsOIbQ"
   },
   "source": [
    "It's now time for training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "h3b2BNJUKKas",
    "outputId": "e3c8b8e7-0beb-4479-9829-793f781e1a03"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/model/sort_pooling/map/while/gradients/model/sort_pooling/map/while/GatherV2_grad/Reshape_1:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/model/sort_pooling/map/while/gradients/model/sort_pooling/map/while/GatherV2_grad/Reshape:0\", shape=(None, None), dtype=float32), dense_shape=Tensor(\"gradient_tape/model/sort_pooling/map/while/gradients/model/sort_pooling/map/while/GatherV2_grad/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16/16 [==============================] - 6s 245ms/step - loss: 0.6714 - acc: 0.5812 - val_loss: 0.6219 - val_acc: 0.5958\n",
      "Epoch 2/100\n",
      "16/16 [==============================] - 3s 186ms/step - loss: 0.6134 - acc: 0.6236 - val_loss: 0.6052 - val_acc: 0.6347\n",
      "Epoch 3/100\n",
      "16/16 [==============================] - 3s 201ms/step - loss: 0.6186 - acc: 0.6649 - val_loss: 0.6000 - val_acc: 0.7305\n",
      "Epoch 4/100\n",
      "16/16 [==============================] - 3s 203ms/step - loss: 0.6205 - acc: 0.6911 - val_loss: 0.5909 - val_acc: 0.7216\n",
      "Epoch 5/100\n",
      "16/16 [==============================] - 3s 167ms/step - loss: 0.6032 - acc: 0.7193 - val_loss: 0.5846 - val_acc: 0.7335\n",
      "Epoch 6/100\n",
      "16/16 [==============================] - 3s 197ms/step - loss: 0.6057 - acc: 0.7052 - val_loss: 0.5796 - val_acc: 0.7335\n",
      "Epoch 7/100\n",
      "16/16 [==============================] - 3s 202ms/step - loss: 0.6005 - acc: 0.7119 - val_loss: 0.5742 - val_acc: 0.7335\n",
      "Epoch 8/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5962 - acc: 0.7267 - val_loss: 0.5703 - val_acc: 0.7335\n",
      "Epoch 9/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.6131 - acc: 0.7056 - val_loss: 0.5680 - val_acc: 0.7335\n",
      "Epoch 10/100\n",
      "16/16 [==============================] - 3s 205ms/step - loss: 0.6084 - acc: 0.6909 - val_loss: 0.5658 - val_acc: 0.7305\n",
      "Epoch 11/100\n",
      "16/16 [==============================] - 3s 205ms/step - loss: 0.5839 - acc: 0.7146 - val_loss: 0.5641 - val_acc: 0.7305\n",
      "Epoch 12/100\n",
      "16/16 [==============================] - 3s 205ms/step - loss: 0.5530 - acc: 0.7621 - val_loss: 0.5617 - val_acc: 0.7305\n",
      "Epoch 13/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5661 - acc: 0.7350 - val_loss: 0.5630 - val_acc: 0.7305\n",
      "Epoch 14/100\n",
      "16/16 [==============================] - 3s 201ms/step - loss: 0.5630 - acc: 0.7337 - val_loss: 0.5605 - val_acc: 0.7335\n",
      "Epoch 15/100\n",
      "16/16 [==============================] - 3s 192ms/step - loss: 0.6016 - acc: 0.6951 - val_loss: 0.5621 - val_acc: 0.7305\n",
      "Epoch 16/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5817 - acc: 0.7151 - val_loss: 0.5581 - val_acc: 0.7335\n",
      "Epoch 17/100\n",
      "16/16 [==============================] - 3s 169ms/step - loss: 0.5946 - acc: 0.7046 - val_loss: 0.5586 - val_acc: 0.7305\n",
      "Epoch 18/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5751 - acc: 0.7019 - val_loss: 0.5566 - val_acc: 0.7305\n",
      "Epoch 19/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5825 - acc: 0.6985 - val_loss: 0.5564 - val_acc: 0.7335\n",
      "Epoch 20/100\n",
      "16/16 [==============================] - 3s 195ms/step - loss: 0.5571 - acc: 0.7305 - val_loss: 0.5557 - val_acc: 0.7335\n",
      "Epoch 21/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.5840 - acc: 0.7179 - val_loss: 0.5564 - val_acc: 0.7335\n",
      "Epoch 22/100\n",
      "16/16 [==============================] - 3s 201ms/step - loss: 0.5901 - acc: 0.7028 - val_loss: 0.5566 - val_acc: 0.7335\n",
      "Epoch 23/100\n",
      "16/16 [==============================] - 3s 203ms/step - loss: 0.5664 - acc: 0.7113 - val_loss: 0.5555 - val_acc: 0.7335\n",
      "Epoch 24/100\n",
      "16/16 [==============================] - 3s 197ms/step - loss: 0.5623 - acc: 0.7086 - val_loss: 0.5568 - val_acc: 0.7395\n",
      "Epoch 25/100\n",
      "16/16 [==============================] - 3s 178ms/step - loss: 0.5696 - acc: 0.7281 - val_loss: 0.5557 - val_acc: 0.7365\n",
      "Epoch 26/100\n",
      "16/16 [==============================] - 3s 190ms/step - loss: 0.5711 - acc: 0.7219 - val_loss: 0.5561 - val_acc: 0.7365\n",
      "Epoch 27/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.5785 - acc: 0.7109 - val_loss: 0.5567 - val_acc: 0.7335\n",
      "Epoch 28/100\n",
      "16/16 [==============================] - 3s 193ms/step - loss: 0.5649 - acc: 0.7187 - val_loss: 0.5554 - val_acc: 0.7365\n",
      "Epoch 29/100\n",
      "16/16 [==============================] - 3s 195ms/step - loss: 0.5607 - acc: 0.7198 - val_loss: 0.5541 - val_acc: 0.7365\n",
      "Epoch 30/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.5726 - acc: 0.7056 - val_loss: 0.5543 - val_acc: 0.7335\n",
      "Epoch 31/100\n",
      "16/16 [==============================] - 3s 168ms/step - loss: 0.5425 - acc: 0.7476 - val_loss: 0.5552 - val_acc: 0.7335\n",
      "Epoch 32/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5653 - acc: 0.7185 - val_loss: 0.5560 - val_acc: 0.7365\n",
      "Epoch 33/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5708 - acc: 0.6982 - val_loss: 0.5549 - val_acc: 0.7395\n",
      "Epoch 34/100\n",
      "16/16 [==============================] - 3s 192ms/step - loss: 0.5508 - acc: 0.7366 - val_loss: 0.5558 - val_acc: 0.7335\n",
      "Epoch 35/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5567 - acc: 0.7226 - val_loss: 0.5562 - val_acc: 0.7395\n",
      "Epoch 36/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5679 - acc: 0.7013 - val_loss: 0.5549 - val_acc: 0.7395\n",
      "Epoch 37/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5892 - acc: 0.6984 - val_loss: 0.5564 - val_acc: 0.7365\n",
      "Epoch 38/100\n",
      "16/16 [==============================] - 3s 192ms/step - loss: 0.5420 - acc: 0.7372 - val_loss: 0.5562 - val_acc: 0.7395\n",
      "Epoch 39/100\n",
      "16/16 [==============================] - 3s 177ms/step - loss: 0.5589 - acc: 0.7181 - val_loss: 0.5574 - val_acc: 0.7365\n",
      "Epoch 40/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5576 - acc: 0.7301 - val_loss: 0.5589 - val_acc: 0.7305\n",
      "Epoch 41/100\n",
      "16/16 [==============================] - 3s 189ms/step - loss: 0.5383 - acc: 0.7187 - val_loss: 0.5578 - val_acc: 0.7335\n",
      "Epoch 42/100\n",
      "16/16 [==============================] - 3s 205ms/step - loss: 0.5446 - acc: 0.7320 - val_loss: 0.5608 - val_acc: 0.7246\n",
      "Epoch 43/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5459 - acc: 0.7323 - val_loss: 0.5579 - val_acc: 0.7275\n",
      "Epoch 44/100\n",
      "16/16 [==============================] - 3s 187ms/step - loss: 0.5364 - acc: 0.7555 - val_loss: 0.5630 - val_acc: 0.7246\n",
      "Epoch 45/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.5340 - acc: 0.7444 - val_loss: 0.5594 - val_acc: 0.7216\n",
      "Epoch 46/100\n",
      "16/16 [==============================] - 3s 164ms/step - loss: 0.5463 - acc: 0.7311 - val_loss: 0.5599 - val_acc: 0.7216\n",
      "Epoch 47/100\n",
      "16/16 [==============================] - 3s 199ms/step - loss: 0.5320 - acc: 0.7488 - val_loss: 0.5602 - val_acc: 0.7156\n",
      "Epoch 48/100\n",
      "16/16 [==============================] - 3s 201ms/step - loss: 0.5442 - acc: 0.7300 - val_loss: 0.5605 - val_acc: 0.7156\n",
      "Epoch 49/100\n",
      "16/16 [==============================] - 3s 192ms/step - loss: 0.5550 - acc: 0.7249 - val_loss: 0.5607 - val_acc: 0.7186\n",
      "Epoch 50/100\n",
      "16/16 [==============================] - 3s 197ms/step - loss: 0.5388 - acc: 0.7335 - val_loss: 0.5610 - val_acc: 0.7186\n",
      "Epoch 51/100\n",
      "16/16 [==============================] - 3s 194ms/step - loss: 0.5454 - acc: 0.7272 - val_loss: 0.5607 - val_acc: 0.7186\n",
      "Epoch 52/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5828 - acc: 0.6841 - val_loss: 0.5604 - val_acc: 0.7186\n",
      "Epoch 53/100\n",
      "16/16 [==============================] - 3s 194ms/step - loss: 0.5639 - acc: 0.7234 - val_loss: 0.5633 - val_acc: 0.7156\n",
      "Epoch 54/100\n",
      "16/16 [==============================] - 3s 189ms/step - loss: 0.5249 - acc: 0.7308 - val_loss: 0.5594 - val_acc: 0.7156\n",
      "Epoch 55/100\n",
      "16/16 [==============================] - 3s 184ms/step - loss: 0.5386 - acc: 0.7341 - val_loss: 0.5606 - val_acc: 0.7096\n",
      "Epoch 56/100\n",
      "16/16 [==============================] - 3s 194ms/step - loss: 0.5322 - acc: 0.7389 - val_loss: 0.5581 - val_acc: 0.7246\n",
      "Epoch 57/100\n",
      "16/16 [==============================] - 3s 168ms/step - loss: 0.5179 - acc: 0.7369 - val_loss: 0.5588 - val_acc: 0.7156\n",
      "Epoch 58/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5261 - acc: 0.7462 - val_loss: 0.5604 - val_acc: 0.7096\n",
      "Epoch 59/100\n",
      "16/16 [==============================] - 3s 201ms/step - loss: 0.5368 - acc: 0.7270 - val_loss: 0.5615 - val_acc: 0.7066\n",
      "Epoch 60/100\n",
      "16/16 [==============================] - 3s 193ms/step - loss: 0.5241 - acc: 0.7480 - val_loss: 0.5640 - val_acc: 0.7036\n",
      "Epoch 61/100\n",
      "16/16 [==============================] - 3s 191ms/step - loss: 0.5473 - acc: 0.7219 - val_loss: 0.5607 - val_acc: 0.7186\n",
      "Epoch 62/100\n",
      "16/16 [==============================] - 3s 195ms/step - loss: 0.5362 - acc: 0.7281 - val_loss: 0.5621 - val_acc: 0.7006\n",
      "Epoch 63/100\n",
      "16/16 [==============================] - 3s 197ms/step - loss: 0.5136 - acc: 0.7522 - val_loss: 0.5626 - val_acc: 0.6976\n",
      "Epoch 64/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5234 - acc: 0.7319 - val_loss: 0.5599 - val_acc: 0.7186\n",
      "Epoch 65/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5479 - acc: 0.7350 - val_loss: 0.5578 - val_acc: 0.6976\n",
      "Epoch 66/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5199 - acc: 0.7446 - val_loss: 0.5567 - val_acc: 0.7096\n",
      "Epoch 67/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5580 - acc: 0.7164 - val_loss: 0.5566 - val_acc: 0.7186\n",
      "Epoch 68/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5338 - acc: 0.7511 - val_loss: 0.5653 - val_acc: 0.7156\n",
      "Epoch 69/100\n",
      "16/16 [==============================] - 3s 176ms/step - loss: 0.5481 - acc: 0.7079 - val_loss: 0.5557 - val_acc: 0.7126\n",
      "Epoch 70/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.5231 - acc: 0.7443 - val_loss: 0.5552 - val_acc: 0.7036\n",
      "Epoch 71/100\n",
      "16/16 [==============================] - 3s 197ms/step - loss: 0.5372 - acc: 0.7386 - val_loss: 0.5601 - val_acc: 0.7036\n",
      "Epoch 72/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5292 - acc: 0.7451 - val_loss: 0.5597 - val_acc: 0.7126\n",
      "Epoch 73/100\n",
      "16/16 [==============================] - 3s 193ms/step - loss: 0.5164 - acc: 0.7835 - val_loss: 0.5580 - val_acc: 0.7066\n",
      "Epoch 74/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5153 - acc: 0.7473 - val_loss: 0.5532 - val_acc: 0.7126\n",
      "Epoch 75/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5090 - acc: 0.7411 - val_loss: 0.5565 - val_acc: 0.7096\n",
      "Epoch 76/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5250 - acc: 0.7493 - val_loss: 0.5597 - val_acc: 0.7186\n",
      "Epoch 77/100\n",
      "16/16 [==============================] - 3s 191ms/step - loss: 0.5065 - acc: 0.7502 - val_loss: 0.5534 - val_acc: 0.7126\n",
      "Epoch 78/100\n",
      "16/16 [==============================] - 3s 176ms/step - loss: 0.5329 - acc: 0.7478 - val_loss: 0.5596 - val_acc: 0.7275\n",
      "Epoch 79/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5188 - acc: 0.7462 - val_loss: 0.5564 - val_acc: 0.7066\n",
      "Epoch 80/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5044 - acc: 0.7632 - val_loss: 0.5582 - val_acc: 0.7126\n",
      "Epoch 81/100\n",
      "16/16 [==============================] - 3s 198ms/step - loss: 0.5136 - acc: 0.7506 - val_loss: 0.5565 - val_acc: 0.7216\n",
      "Epoch 82/100\n",
      "16/16 [==============================] - 3s 189ms/step - loss: 0.5103 - acc: 0.7625 - val_loss: 0.5613 - val_acc: 0.7186\n",
      "Epoch 83/100\n",
      "16/16 [==============================] - 3s 197ms/step - loss: 0.4960 - acc: 0.7588 - val_loss: 0.5613 - val_acc: 0.7156\n",
      "Epoch 84/100\n",
      "16/16 [==============================] - 3s 182ms/step - loss: 0.5253 - acc: 0.7344 - val_loss: 0.5563 - val_acc: 0.7305\n",
      "Epoch 85/100\n",
      "16/16 [==============================] - 3s 179ms/step - loss: 0.5198 - acc: 0.7532 - val_loss: 0.5635 - val_acc: 0.7246\n",
      "Epoch 86/100\n",
      "16/16 [==============================] - 3s 202ms/step - loss: 0.4844 - acc: 0.7765 - val_loss: 0.5619 - val_acc: 0.7305\n",
      "Epoch 87/100\n",
      "16/16 [==============================] - 3s 193ms/step - loss: 0.4879 - acc: 0.7859 - val_loss: 0.5570 - val_acc: 0.7246\n",
      "Epoch 88/100\n",
      "16/16 [==============================] - 3s 207ms/step - loss: 0.4751 - acc: 0.7795 - val_loss: 0.5553 - val_acc: 0.7275\n",
      "Epoch 89/100\n",
      "16/16 [==============================] - 3s 190ms/step - loss: 0.5132 - acc: 0.7470 - val_loss: 0.5590 - val_acc: 0.7365\n",
      "Epoch 90/100\n",
      "16/16 [==============================] - 3s 191ms/step - loss: 0.4994 - acc: 0.7593 - val_loss: 0.5579 - val_acc: 0.7305\n",
      "Epoch 91/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5036 - acc: 0.7601 - val_loss: 0.5637 - val_acc: 0.7335\n",
      "Epoch 92/100\n",
      "16/16 [==============================] - 3s 165ms/step - loss: 0.5069 - acc: 0.7554 - val_loss: 0.5546 - val_acc: 0.7335\n",
      "Epoch 93/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.4804 - acc: 0.7698 - val_loss: 0.5612 - val_acc: 0.7455\n",
      "Epoch 94/100\n",
      "16/16 [==============================] - 3s 196ms/step - loss: 0.5093 - acc: 0.7378 - val_loss: 0.5576 - val_acc: 0.7335\n",
      "Epoch 95/100\n",
      "16/16 [==============================] - 3s 200ms/step - loss: 0.4972 - acc: 0.7593 - val_loss: 0.5542 - val_acc: 0.7485\n",
      "Epoch 96/100\n",
      "16/16 [==============================] - 3s 202ms/step - loss: 0.4959 - acc: 0.7905 - val_loss: 0.5664 - val_acc: 0.7455\n",
      "Epoch 97/100\n",
      "16/16 [==============================] - 3s 180ms/step - loss: 0.5301 - acc: 0.7613 - val_loss: 0.5575 - val_acc: 0.7246\n",
      "Epoch 98/100\n",
      "16/16 [==============================] - 3s 190ms/step - loss: 0.4985 - acc: 0.7727 - val_loss: 0.5641 - val_acc: 0.7305\n",
      "Epoch 99/100\n",
      "16/16 [==============================] - 3s 191ms/step - loss: 0.4766 - acc: 0.7738 - val_loss: 0.5578 - val_acc: 0.7275\n",
      "Epoch 100/100\n",
      "16/16 [==============================] - 3s 195ms/step - loss: 0.5121 - acc: 0.7636 - val_loss: 0.5636 - val_acc: 0.7305\n"
     ]
    }
   ],
   "source": [
    "epochs = 100\n",
    "history = model.fit(\n",
    "    train_gen, epochs=epochs, verbose=1, validation_data=test_gen, shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gdPBykJ4KPrV"
   },
   "outputs": [],
   "source": [
    "# https://stellargraph.readthedocs.io/en/stable/demos/graph-classification/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T1lM0v05_zJe"
   },
   "source": [
    "## Supervised node representation learning using GraphSAGE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "id": "gERK1Zen_xL7",
    "outputId": "5959207e-3139-4262-fdbf-502d255bc826"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "The Cora dataset consists of 2708 scientific publications classified into one of seven classes. The citation network consists of 5429 links. Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary. The dictionary consists of 1433 unique words."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from stellargraph import datasets\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "dataset = datasets.Cora()\n",
    "display(HTML(dataset.description))\n",
    "G, nodes = dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xrkhfxtenQ4i"
   },
   "source": [
    "Let's split the dataset into training and testing set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "RZsS_u7v_5vc"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train_nodes, test_nodes = train_test_split(\n",
    "    nodes, train_size=0.1, test_size=None, stratify=nodes\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mm4-Me5GnVce"
   },
   "source": [
    "Since we are performing a categorical classification, it is useful to represent each categorical label in its one-hot encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "dP-sXgekAFOY"
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing, feature_extraction, model_selection\n",
    "label_encoding = preprocessing.LabelBinarizer()\n",
    "train_labels = label_encoding.fit_transform(train_nodes)\n",
    "test_labels = label_encoding.transform(test_nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KJSpM4cnnfUN"
   },
   "source": [
    "It's now time for creating the mdoel. It will be composed by two GraphSAGE layers followed by a Dense layer with softmax activation for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "BP5G49akANxy"
   },
   "outputs": [],
   "source": [
    "from stellargraph.mapper import GraphSAGENodeGenerator\n",
    "batchsize = 50\n",
    "n_samples = [10, 5, 7]\n",
    "generator = GraphSAGENodeGenerator(G, batchsize, n_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "qkgF2VvWAlct"
   },
   "outputs": [],
   "source": [
    "from stellargraph.layer import GraphSAGE\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "graphsage_model = GraphSAGE(\n",
    "    layer_sizes=[32, 32, 16], generator=generator, bias=True, dropout=0.6,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "S_g_yOhjAovl"
   },
   "outputs": [],
   "source": [
    "gnn_inp, gnn_out = graphsage_model.in_out_tensors()\n",
    "outputs = Dense(units=train_labels.shape[1], activation=\"softmax\")(gnn_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "HeMlOuDnA9B_"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.losses import categorical_crossentropy\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "model = Model(inputs=gnn_inp, outputs=outputs)\n",
    "model.compile(optimizer=Adam(lr=0.003), loss=categorical_crossentropy, metrics=[\"acc\"],)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MqF4EWFbnwU9"
   },
   "source": [
    "We will use the flow function of the generator for feeding the model with the train and the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "x-5xmzRqBDCX"
   },
   "outputs": [],
   "source": [
    "train_gen = generator.flow(train_nodes.index, train_labels, shuffle=True)\n",
    "test_gen = generator.flow(test_nodes.index, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "952-5V6Xn45o"
   },
   "source": [
    "Finally, let's train the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sS3vnQ_HBZxK"
   },
   "outputs": [],
   "source": [
    "history = model.fit(train_gen, epochs=20, validation_data=test_gen, verbose=2, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fGJXHirZBcuL"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iFZJ2OJcoQgi"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Supervised_GraphML.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
